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Introducing a New Learning and Teaching Evaluation Planning 
Framework for Small Internally Funded Projects in Higher Education 


Abstract 

Scholarly evaluation practices in learning and teaching projects are under-reported in the literature. In order 
for robust evaluative measures to be implemented, a project requires a well-designed evaluation plan. This 
research study describes the development of a practical evaluation planning framework through an action 
research approach, using primary data from learning and teaching practitioners in higher education. 
Underpinned by the literature, this framework is accessible to a wide range of disciplinary scholars and is 
accompanied by a set of recommendations that together contribute to the improvement of the evaluation of 
learning and teaching innovations in higher education. 
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Introduction 

In the higher-education sector, small grants, also known as internal or local funding, are one way 
to support projects that investigate the effects of an intervention on learning and teaching 
outcomes (Clark & Hall 2008). Such projects and investigations have the potential to instigate the 
start of a change process that brings ideas, concepts and implementation to a wider audience 
through associated dissemination activities such as presentations and written publications 
(Gannaway et al. 2013; Hinton 2014). 

Similarly, processes of formative evaluation (Cronbach 1982) can yield findings that can 
immediately affect the project or intervention design and lead to improved project outcomes 
(Harris, Jones & Coutts 2010; Patton 1994). Evaluation is often mandated by external funding 
organisations (Chesterton & Cummings 2011; Rog 2015). However, smaller grants (which can 
start from as little as $1,000 but tend to be capped around $10,000) offered by internal funders in 
Australian universities do not always specify evaluative requirements, other than mandating that 
evaluation must be carried out; and those that are more specific often do not follow up or 
implement measures to ensure that evaluation is enacted (Huber & Harvey 2013). 

Scholarly evaluation practices in learning and teaching projects are under-reported in the literature 
(Alexander 1999; Christie 2003; Smith 1993). The reasons for this are unclear; however, some 
emerging research suggests that evaluation is influenced by a project leader’s conception of 
evaluation as well as the lack of resources (in terms of both time and budget) to implement 
evaluative measures (Huber & Harvey 2016a; Rog 2015; Worthern & Sanders 2011). Provision of 
resources to support evaluation can also influence engagement with learning and teaching 
evaluation (Smeal, Southwell & Locke 2011). 

For robust evaluative measures to be implemented, a project requires a well-designed evaluation 
plan. Studies have found that it is often in the planning stage that evaluation goes awry (Brandon 
1998; Nesman, Batsche & Hernandez 2007). Alongside mandatory evaluation requirements and 
budget allocation, there are a variety of resources to support project teams with implementation 
evaluation (for example, BetterEvaluation n.d.; Chesterton & Cummings 2011; JISC n.d.; Oliver et 
al. 2002). Such resources are expansive and tend to be aimed at large projects, such as those that 
are usually externally funded and run for periods of 12 to 24 months. For a detailed overview of 
the international contexts for university funding of learning and teaching enhancements, see Huber 
and Harvey (2013). 

But what of smaller projects with limited resources for both time and money? Current evaluation 
resources designed for large-scale projects can be seen as inappropriate, as they are too complex 
for small projects. For example, one evaluation toolkit developed as part of a project funded by the 
UK Joint Information Systems Committee (JISC n.d.) took 4.5 hours to complete. Whilst this was 
reasonable for the complex demands of evaluating large projects, it was considered to be ill-suited 
to smaller studies (Oliver et al. 2002). Similarly, findings have been reported about the 
educational-evaluation literature: “although an enormous body of literature around evaluation has 
been developed, teachers in the trenches trying to bring about reforms are mostly unfamiliar with 
this literature” (Hannah 1996, p. 412). Whilst resources and information on evaluation are 
available, they are not widely used. In a study of program-evaluation practitioners and evaluation 
theorists, the author found that “the gap between the common evaluator and the notions of 
evaluation practice put forth by academic theorists has yet to be bridged” (Christie 2003, p. 34). 
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At the same time, it must not be assumes that all smaller projects have similar evaluation needs 
(Huber & Harvey 2016b; Oliver et al. 2002; Stoner et al. 2012). However, the importance of 
evaluation and its role for improved project outcomes (leading to innovation and change), no 
matter the project size, cannot be underestimated. A precursor study to this one investigated 
whether the evaluation practices of small-scale project leaders were based on particular forms or 
approaches (Owen 2010). Findings indicated that only 27% of the participants used a named 
evaluation approach, and that these were based on prior experience (Huber & Harvey 2016a). 
Novices in the educational-evaluation space, who are often the recipients of small-scale project 
funding, did not align with any particular theory or framework. 

The aim of this research study was to develop a practical evaluation-planning framework, 
accessible to a wide range of disciplinary scholars that would contribute to the improvement of the 
evaluation of learning and teaching innovations in higher education. The research question 
addressed by this study is “What is required to develop a framework to support the evaluation of 
small-scale internally funded learning and teaching projects?” This study is significant because 
findings can contribute to the improvement of the evaluation of small-scale learning and teaching 
innovations in higher education. Empirical evidence on the practices of evaluation scholars has 
been called for (Christie 2003; Rog 2015; Smith 1993), and this planning framework can 
contribute to the collection of such data. 

This paper describes the development of a new evidence-based framework for evaluating small 
learning and teaching projects. The background section will situate this research within a wider 
study and summarise the previous findings. The methodology used in this study is introduced and 
methods used to develop and trial the framework are presented. The findings from this research 
are integrated with earlier findings and are discussed in the context of emerging evaluation 
practitioners in the higher-education sector. 


Background 

This paper describes the development of an evidence-based evaluation-planning framework 
designed for small, internally funded learning and teaching projects, through an action-research 
approach (Coghlan & Brannick 2010; McNiff 2001). The purpose of the framework is to help 
project teams develop a flexible evaluation plan that is relevant to their contextual needs. This 
study is premised on the assumption that a well-designed evaluation plan can lead to more-focused 
evaluation outcomes, which in turn can make a significant contribution to the adoption (or not) of 
any learning and teaching innovations. 

The evaluation-planning framework introduced here is informed by leading evaluative scholars 
and is predominantly based on the work of Paul Chesterton and Rick Cummings (2011), who in 
turn applied the work of Owen (2006) to develop resources for the Australian Office for Learning 
& Teaching (OLT) and their predecessor, the Australian Learning & Teaching Council (ALTC). 
Their resources were designed to support grant applicants in their evaluation of large (upwards of 
$50,000), government-funded learning and teaching projects in higher education. The framework 
described in this paper also incorporates the importance of developing “Key Evaluation 
Questions” that can be adequately answered (Datta 1997); the use of evaluation (Patton 1994); 
stakeholder engagement (Stufflebeam 2011); identification of evaluation-study audiences 
(Saunders 2000); identification of the level of generalisability for further application of evaluation 
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findings (Scriven 1991); and the determination of the scope of success factors and, conversely, the 
points at which a project may need to be stopped (Phillips, McNaught & Kennedy 2012). The 
focus for this new framework is small learning and teaching projects. The framework is given the 
name SPELT (small-project evaluation in learning and teaching). 

The design of the SPELT framework was the outcome of a series of precursor studies and phases 
(Figure 1). The first study investigated the evaluative practice of a cohort of learning and teaching 
grant holders (n=15) in a higher-education context (Huber & Harvey 2016a). Thematic analysis 
was used to analyse the transcripts of interviews with 15 project leaders. This resulted in four key 
themes being drawn from the data: how evaluation is conceptualised, particularly the overlap, even 
conflation, between evaluation and research; capability building within the sector; resourcing in 
terms of time and money; and the role of an action-oriented approach to evaluation. It was found 
that whilst a number of resources existed to support the evaluation of learning and teaching 
projects, they were rarely known or used. Furthermore, the study found that evaluation theory and 
the practice of project evaluation were misaligned, and that this relationship can be inhibited by a 
project leader’s perception of evaluation. 



The focus of this paper 



f ~ \ 

What specific strategies 
can be used to overcome 
barriers to successful 
evaluation 
implementation? 


Figure 1. The focus of this paper, embedded within the three phases of this research study 
[there is an odd little mark in the box that starts “How does a project leader’s perception”] 

A second study built on the findings of the first phase and used case-study methodology to 
investigate three internally funded learning and teaching projects in greater depth (Figure 1). 
These case examples provided the data for analysis using an explanatory approach, and the 
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projects shared a focus on curriculum design in arts and humanities disciplines at an Australian 
university. The researchers acted in the role of participant-as-observer (Babchuk 1962) within each 
project and investigated the contextual factors that influenced its evaluation practices (Huber & 
Harvey 2016b). This second phase found that factors influencing the praxis of evaluation include 
timeframe; previous experience of leading a project; and the mandating (or lack thereof) of 
evaluation from the projects’ grant-funding body. Informed by the research findings, four 
strategies to enhance the adoption of systematic evaluation in small learning and teaching projects 
were discussed: making evaluation-support mechanisms more explicit; the development of an 
evaluation community of practice with other grant recipients that could enable engagement in 
collaborative reflection; a requirement for grant applicants to identify how their project builds on 
previous work; and the need for a flexible framework for evaluation planning. Figure 1 gives a 
visual overview of the three study phases. 

During the first two phases prior to this study, the interview protocols used in both data-collection 
stages were developed and refined based on a synthesis of seminal research by evaluation scholars 
(Huber & Harvey 2016a; 2016b). Using the protocols, findings and feedback from the participants 
in these first two phases, 12 questions were sequenced into a proposed framework. Findings from 
the previous phases of the study indicated a lack of clarity and some misconceptions about how 
and what to evaluate, so the framework became a series of reflective prompts to aid the novice 
evaluator. Table 1 summarises the questions that form the proposed framework, along with the 
corresponding influential literature/evaluation scholar. 

Table 1.12-Step evaluation-planning framework 


Step 

Question 

Literature 

1 . 

What is the purpose and scope of the 
evaluation? 

Chesterton & Cummings (2011); 

Saunders (2000); Stufflebeam (2011) 

2. 

How will the information from the 
evaluation be used? 

Owen (2006); Patton (1994); Saunders 
(2000); Scriven (1991) 

3. 

Who are the stakeholders of the project 
and the evaluation? 

Chesterton & Cummings (2011); 
Stufflebeam (2011) 

4. 

Who is the study audience for the 
evaluation results? 

Saunders (2000) 

5. 

5.1 

What are the Key Evaluation 

Questions? 

Can they be answered adequately? 

Chesterton & Cummings (2011) 

Datta (1997) 

6. 

What data and evidence will be 
collected? 

Saunders (2000) 

7. 

How will the data be analysed? 

Chesterton & Cummings (2011); Owen 
(2006) 

8. 

What are the criteria for judgement? 

Chesterton & Cummings (2011); Owen 
(2006) 

9 . 

Who will review the evaluation plan? 

Chesterton & Cummings (2011); 

Saunders (2000); Scriven (1991) 

10. 

What amount of generalisability will 
there likely be? 

Scriven (1991) 

11. 

What reporting strategies will be used? 

Chesterton & Cummings (2011); 
Stufflebeam (2011) 
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12 . 


What are the critical success factors? Is 
there an exit strategy in place? 


Phillips, McNaught & Kennedy (2012) 


This current study (Figure 1, Phase 3) builds on the first and second phases, and begins with the 
research question “What is required to develop a framework to support the evaluation of small, 
internally funded learning and teaching projects?” This paper reports on the development of the 
framework. 


Methodology 

The process of developing this new SPELT framework was framed within an action-research 
approach (Coghlan & Brannick 2010; McNiff 2001). Action research “seeks to bring together 
action and reflection, theory and practice, in participation with others, in the pursuit of practical 
solutions to issues of pressing concern...” (Reason & Bradbury 2001 p. 1). In this case, the issue 
was the practice of evaluation in small learning and teaching projects, and the practical solution 
was a framework that would be flexible and useable. Action research also requires iterative cycles 
of action, observation, evaluation and redesign (Wadsworth 2011). This phase of the study 
investigates the design and development of a framework through two iterative cycles of reflection 
on practice, feedback and redesign. The ultimate output of this research was to produce an 
evaluation-planning framework that could support educators on the journey to improving their 
learning and teaching practice. 

Action research was enacted through the two sessions with learning and teaching project- 
evaluation practitioners. Participants reflected on their practice by interacting with the framework. 
The component parts of this session are shown in Figure 2. Then the researcher redesigned the 
framework based on feedback and observations. Each cycle will now be described. 


Part 1. Two 
individual 
reflection 
questions 



Part 2. Two 
small groups 
workshop the 
evaluation 
planning 
framework 
noting down 
examples for 
each step 


Part 3. Whole 
cohort focus 
group. 

Recorded and 
transcribed 



Part 4. Final 
individual 
reflection 
question 


Figure 2. Component parts of the focus-group session 

Action-research cycle one 

Participants 

The initial 12-step evaluation-planning framework (Table 1) was piloted with a group of 
academics from one Australian metropolitan university. For the first round of investigations, 
invitations to participate in the study were sent to all staff who had received an internal learning 
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and teaching grant during the year of the study (n=33). In this invitation it was explained that the 
purpose of the framework was to assist participants with their small-scale project. Seven staff 
responded and became the focus-group participants. These participants included two males and 
five females from the following disciplines: journalism, communications, education, accounting, 
civil engineering, environmental engineering and nursing. Two of the participants had also 
previously received large-scale, externally funded learning and teaching grants; one was new to 
learning and teaching grants; and others had received numerous small-scale, internally funded 
learning and teaching grants. 

Method 

The framework was interrogated by the participants through a hands-on workshop and focus group 
(combined). This one-hour session was broken into four parts (Figure 2). The first part required 
participants to individually reflect on their evaluative practice in small-scale learning and teaching 
projects and write down their answers to two questions: 1. What does evaluation mean to you? 

And 2. How do you feel about evaluation? This was done to focus the group on the topic and offer 
them time for critical self-reflection (Mezirow 1998). In the second part of the session (Figure 2, 
part 2), participants worked in two small groups to workshop the evaluation-planning framework 
(Table 1). This was considered the “act” part of the action research cycle. They were encouraged 
to write notes and examples against each of the steps. The human-ethics consent received for this 
project authorised the collection of participant’s written notes. These notes have since been used to 
develop a supporting resource for staff that contains specific, practical and authentic examples to 
illustrate each of the steps in the evaluation-planning framework. Appendix A contains an excerpt. 

Table 2. Focus-group questions 


1 . 

What do you like about this framework (what works well)? 

2 . 

What parts of this framework do you think need improvement? 

3. 

Is there anything missing from this framework? 

4. 

Is the framework practical? If not, what is needed to make it more so? 

5. 

Could you start to evaluate your project once you have completed the 
framework template? 


The third part of the session (Figure 2, part 3) involved a focus group to unpack the feedback on 
the evaluation-planning framework from the participants. This contributed to the “evaluate” part of 
the action-research cycle. The focus-group questions answered in the group context are listed in 
Table 2. The focus-group component was recorded and transcribed as verbatim text. In the final 
part of this session (Figure 2, part 4), participants were asked to answer a third question 
(individually): “Has your perception of evaluation changed during this session?” This data was 
collected as an evaluative measure for the planning framework. 

Data analysis 

All data from the session (reflective answers, examples, transcripts) were triangulated with 
relevant evaluative and scholarly learning and teaching literature to form a rich, robust and 
comprehensive validated account is (Cohen & Crabtree 2006). Together with the findings from the 
first and second phases of the wider study (Figure 1), this data informed the development of a 
second iteration of the evaluation-planning framework (Table 3). This iteration contained only six 
steps; the reasons for this will be described in the following results and discussion section. 
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Action-research cycle two 

Participants 

This second iteration of the evaluation-planning framework was trialled with a new cohort of 
participants. Invitations were sent to all learning and teaching grant awardees at one large, 
metropolitan university (as previously but without the participants from cycle 1). Seven people 
responded: six females and one male from the following disciplines: international studies, 
biological sciences, mathematics, civil engineering, project management and education. Four of 
the participants had previously received both large-scale, externally funded and small-scale, 
internally funded learning and teaching grants; two were new to the learning and teaching grant 
space. 

Method 

The session for the second cohort was run exactly as the first cycle, except that this time the 
evaluation-planning framework only contained six steps (Table 3). The examples, which had been 
provided by the participants of the first cycle (Figure 2, part 2) and which described authentic 
applications of each step of the evaluation-planning framework (Table 1), were also used in the 
second cycle. This helped to situate the framework and questions in the context of higher 
education (O’Neill 1995). Participants were also reminded that the focus of this planning 
framework was on small-scale learning and teaching projects. 

Data analysis 

The transcribed data was again triangulated with the evaluation literature and the findings from 
phase 1 of the wider study (that whilst a number of resources exist to support evaluation of 
learning and teaching projects, they are rarely known or used); findings from phase 2 of the wider 
study (that the project leaders’ conception and experience of evaluation influences their praxis); 
and findings from the first cycle cohort, who had used the 12-step framework (as detailed in the 
following section). This was to determine whether any further changes were required to produce 
the final evaluation-planning framework. 


Findings and discussion 

In this section comments from the participants in the two focus groups are used to highlight key 
findings. The participants are designated according to their focus group (cycle 1 or 2) and their 
discipline (letters). The section uses the process of “enfolding” (Eisenhardt 1989), whereby the 
evaluation literature is used to interpret, explain and substantiate the relevance of the participants’ 
comments. Participants’ comments are also used throughout the following section to highlight and 
explain research findings. 

Cycle one: 12-step framework 

The participants in the first focus group were introduced to the 12-step evaluation-planning 
framework (Table 1) through instructor-led discussion of each of the steps. Participants were 
encouraged to ask questions and share their practice with relevant examples. There was a general 
openness to talking about evaluation and a positive atmosphere in the room during the discussion, 
particularly when talking about their own projects. This aligns with a previous study of small- 
project grant holders (Huber & Harvey 2016a) that indicated that whilst time and energy is 
invested in a project, there are often few (and sometimes no) feedback opportunities, either formal 
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or informal, focusing on evaluation. This is an important requirement if the focus is to encourage 
change or to go one step further and support the leadership of change in learning and teaching 
(Ramsden et al. 2007). 

Previous findings have indicated that learning and teaching evaluation practitioners, particularly 
those new to small-grant projects, require support (Huber & Harvey 2016a). Distributed leadership 
is an approach that could potentially provide this support. In this approach, leadership capacity can 
be developed in an environment of respectful collaboration, with the aim of sustaining 
improvements in teaching and learning (Jones et al. 2014). A key feature of distributed leadership 
is that everyone involved in a process (for example, the project team members) has their strengths 
acknowledged and, where appropriate, leads some part of the project. This aligns with a 
participatory action-research approach, and with the recommendation from phase 1 of the wider 
study that participatory action research could be investigated further for small learning and 
teaching projects. Traditional approaches to evaluation are not participatory. There is a perception 
that evaluators need to be experts, and that they are external to the project, often remaining aloof 
and merely observing, then judging, the project. Some project leaders feel pressure from such 
external evaluation activities and this leads to a negative perception of evaluation (Huber & 
Harvey 2016a). This is not a good fit for the academy. Exemplary evaluation activities could be 
modelled through a distributed-leadership approach, thereby acting as an evaluation-support 
mechanism. 

The cohort of grant recipients participating in the first-cycle session were able to share their 
experiences, and therefore “relive” the process. The notion of learning through a community of 
practice is not new (Wenger 1998). The interaction between focus group participants suggests the 
need for an ongoing conversation around evaluation and findings from innovative projects as can 
happen in a community of practice. This aligns with findings from phase 1 (Huber & Harvey 
2016a) and phase 2 (Huber & Harvey 2016b) that practitioners felt they were working in a 
vacuum, and had no one with whom to share their thoughts and questions. Such a supportive 
environment can provide the foundations needed to facilitate scholarly practice, the 
encouragement needed for organisational leadership to develop at a local level and the opportunity 
to develop peer networks. 

The workshop component of the session (Figure 2, part 2) began with the researcher (who was 
also the presenter) introducing the purpose of the evaluation-planning framework. It was observed 
that the participants required further clarity when discussing the synergies between research and 
evaluation. As a result, it was noted that a diagram might have been useful to help clarify this 
relationship, and that one would be used for the second-cycle focus group (Figure 3). This 
diagram, which was developed based on the work of John Owen (2006), scaffolded the 
participants’ understanding of evaluation and research synergies. One example is highlighted in 
this comment: 


I think step 6 [Table 1] needs to be really clear too, the thing about what data and 
evidence will be collected. I immediately thought of the data about student outcomes 
rather than when you said (( data for when you evaluate the project”, and that's a totally 
different perspective. And that wasn ’t quite clear until you said that (Participant 1A). 


Other opinions emerged around the confusion between evaluation and project reporting; for 
example “There is some confusion in the small teaching and learning grants about what is an 
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evaluation and what is a report” (Participant IN). It would therefore be useful to clarify meanings 
in the final evaluation-planning framework, through participant’s reflection on their own context 
(Botcheva, Shih & Huffman 2009; Schwandt 2014). Similar findings were noted in phase 1 of the 
wider study. A number of concepts were conflated, such as the distinction between research, 
evaluation and project outcomes; and the difference between stakeholders and study audiences 
required clarification (Huber & Harvey 2016a). 

In general, there was a group-wide agreement on the value of evaluation (based on the responses 
to the pre-session question “What does evaluation mean to you?”, and a willingness to embrace it 
in their projects. However, this could be due to the self-nomination of participants, and may not 
necessarily be true of all grant-holders. In phase 1 of the wider study, some of the project leaders 
were hesitant to embrace evaluation, and in fact stated that if evaluation had been mandated they 
wouldn’t have applied for the grant money, as they would have felt that they were being 
“watched” (Huber & Harvey 2016a). 

Two participants felt overwhelmed by the framework because it was too long. This led to 
comments from members of the group regarding “having to” complete it. For example, one said, 
“The motivation to do it properly depends on whether it is useful for yourself. If it’s just ticking 
off boxes for someone else, then the evaluation will be cursory and not very interesting or useful” 
(Participant 1C). This aligns with findings from phase 1 of the wider study and from the literature, 
in that many of the complex evaluation approaches reported are not suitable to smaller-scale 
projects (Stoner et al. 2012). 

Some participants commented on the time it would take to complete the planning framework, and 
said that if this was mandated it would discourage them from doing evaluation. This aligns with 
findings from the literature that lack of time was one reason why projects often did not complete 
evaluation requirements (Harris, Jones & Coutts 2010; Huber & Harvey 2013; Ryan, Chandler & 
Samuels 2007). The SPELT framework developed in this study is intended as a resource to help 
grant holders and project leaders formulate their evaluation plan in as time-efficient way as 
possible. 

Another thread from the discussion around mandating evaluation led to this comment: “I just 
forget to do the evaluation and no one says anything” (Participant 1J). Phase 1 of the wider study 
showed that when funding bodies did not follow up regarding final project reports, the evaluation 
tended to fall by the wayside; this led to a recommendation from that phase of the study that grant 
awardees receive sufficient support to understand the “how” and “what” of evaluation. (Huber & 
Harvey 2016a). 

One participant felt that more explanation was required as to how evaluation results could be 
useful: “So this [framework] doesn’t instil in me a sense of why I should complete it - I don’t get 
a sense that it will help the project outcomes or how its useful for future research” (Participant 1 J). 
Inclusion with the framework of some background text that describes the usefulness of both 
formative and summative evaluation (Scriven 1996) could address this point. Another participant 
reflected on their previous experiences working on projects in a non-academic environment where 
they would ask questions at the end of the project about what worked well and what didn’t. 
However, they felt that this wouldn’t translate to an academic scenario because “that was more 
about individual skill development and project management skills than the actual project itself’ 
(Participant 1 A). However, such learning from a project is indeed valuable if skills can be applied 
again more effectively to the next iteration or a different project (Preskill & Torres 2000). In fact, 
evaluation itself “involves ongoing learning” (Rog 2015, p. 235). 
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M 


negotiating of an evaluation plan 


stage 2l 


implementing of the evaluation design 
producing of evaluation findings 


disseminating of findings to identified audiences 
making/implementing recommendations 


Figure 3. Three stages of the project evaluation cycle (adapted from John Owen 2006) 

One participant (IN) stated that evaluation would be perceived as more useful if it were aligned 
closely with the grant-application questions. This highlights the importance of perception, a topic 
that will be discussed in the next section. It also suggests the importance of context and how it can 
influence decision-making (Tourmen 2009). It has been shown that beginners (in evaluative 
practice) find it difficult to cope with the complexity of evaluative activity, such as not being able 
to control contextual factors, waiting for stakeholder’s decisions and compromising (Tourmen 
2009). A structured framework to assist with an evaluation’s planning could be highly beneficial 
for those new to evaluation. Defining the purpose of the evaluation (Step 1, Table 1) is imperative 
to successfully deliver outcomes that others may set (Bocheva, Shih & Huffman 2009; Saunders, 
2000; Stufflebeam, 2011) or perceive to be important. 

Modifying the evaluation-planning framework 

Based on the findings from the first cycle and from earlier phases of the wider study, a series of 
changes were made to the framework (the redesign phase of the action-research cycle). There 
existed a perception that it would take too long to complete a 12-step framework. Several steps 
were combined to reduce the number of steps from 12 to six. The main developmental changes are 
listed together with the rationale underpinning the need for each change. 

• Conflation of the terms stakeholders and study audiences caused some confusion. Two 
steps (steps 2 and 3 of Table 1) were combined so that those unsure of the difference 
between these two terms could simply apply the one that would be relevant to their 
context (Table 3, step 2). 

• Collection and analysis of data were combined into one step, as these activities are 
naturally grouped (Figure 3, stage 2). 

• Participants agreed that the evaluations of their internally funded learning and teaching 
projects were small-scale and primarily used formatively for the benefit of the project 
team. In most cases their evaluation plans were not evaluated, and therefore step 9 (Table 
1 ) was removed. 

• Similarly, thoughts about the generalisability of the evaluation were not common in these 
evaluations; this step (Table 1, step 10) was also removed. 

• Critical success factors and exit strategies were areas the participants felt would have 
been discussed in the grant application and therefore was unnecessary for the planning 
framework (removal of step 12). 

• An extra item was added to step 6 (Table 3) to encourage users to reflect on how the 
dissemination strategies could help them. This was in response to participants’ comments 
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about the need to close the quality-enhancement cycle and a similar finding from phase 1 
of the wider study (Huber & Harvey 2016a). 

Table 3. Six-step evaluation-planning framework 


Step 

Question 

1 . 

What is the purpose and scope of the evaluation? 

Consider also how the information will be used. 

2 . 

Who are the stakeholders of the project and of the evaluation? 

3. 

What are the Key Evaluation Questions? 

4. 

What data and evidence will be collected and how will it be analysed? 

5. 

What are the criteria for judgment? 

6 . 

What dissemination strategies will be used and how will this help you? 


Cycle two: Six-step framework 

Participants in the second-cycle workshop were introduced to the modified (six-step) evaluation 
framework (Table 3) and given the opportunity to discuss it in pairs, adding examples from their 
own projects against each step where applicable (Figure 2, part 2). As in the first cycle, which 
used the 12-step framework, there was a general openness to talking about evaluation and a 
positive atmosphere in the room during the discussions, particularly when talking about their 
projects. Again there was a general understanding of the value of evaluation and a willingness to 
embrace it in their projects. The two participants who were new to evaluation, grants and projects 
contributed less to the discussions than the other participants. Nevertheless, throughout the session 
it was observed that they made notes and asked questions of their peers, further exemplifying the 
value and usefulness of a community of practice for learning, development and capacity-building 
in evaluation praxis. For example, some of the more experienced participants could answer 
questions posed by lesser experienced colleagues about terminology in the SPELT framework, 
such as “I found that I didn’t quite understand what you meant by formative and summative.. .with 
regards to evaluation” (participant 2BS). And in response “I thought that the formative was things 
that inform you as you are doing the project. So you are doing a little bit of evaluation as you are 
going through anyway, and you are using the output of that before you complete the project” 
(participant 2S). 

In this group, two of the experienced participants disagreed about the order of the first two 
questions in the planning framework. It is widely agreed that stakeholder involvement strongly 
influences how evaluation findings are used (Harris, Jones & Coutts 2010; Johnson et al. 2009). 
One participant believed that identifying the stakeholders must come first before deciding on the 
scope of the evaluation. I.e. step 2 before step 1. 


Okay, taking a project that I’ve been on. The stakeholders came first. We looked at issues 
they faced and then, okay, so what outcomes are they expecting and how are we going to 
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address them? So, very much, at least from my project point of view, it is the stakeholders 
who are driving the process (Participant 2M). 


However, another participant strongly disagreed. 


I think I would leave the stakeholders where they are and I would leave the outcomes 
front and centre. I like the [framework]. I think it's really helpful to have upfront and to 
he thinking about your whole evaluation at the beginning of the process. For me 
personally, I would be very outcome-focused and make sure Fm really clear on what my 
outcomes are prior to planning (Participant 2PM). 


This second participant’s viewpoint aligns with the guidelines provided by Australian higher- 
education grant funders, the OLT (Chesterton & Cummings 2011) and leading evaluation scholars 
(for example, Alkin 2011; Owen 2006). Stoner et al. (2012) found that the purpose of evaluation 
must be established before the stakeholders are identified. However, the stakeholders can then help 
form the goals and attributes within a project to develop the evaluation framework. 

This suggests the possibility of a flexible starting point for any evaluation framework, or an ability 
to be able to revisit steps according to individual requirements, contexts and stakeholder needs. 

The importance of flexibility to the success of an evaluation was also a main finding in a research 
study on participation (DeLuca, Poth & Searle 2009); the study concluded that evaluators must be 
explicit in how any participatory approach can meet the primary evaluation purpose whilst at the 
same time enhancing use. 

A participant (2PM) commented that they felt another step was needed for the six-step planning 
framework regarding recommendations for the future. For example, users should consider how 
their evaluation results could inform future projects. This was a step that had been removed from 
the initial 12-step version of the framework in an effort to achieve a more parsimonious product. 

As a result of this feedback, it will be reinstated in the final iteration. To minimise the number of 
steps (and therefore the conception of time needed to complete the evaluation), this information 
will be incorporated within the dissemination stage (Table 3, step 6) as a reminder to think through 
future plans. 

Participants in this second-cycle group all agreed that a reminder about the need for ethics 
approval for certain data-collection activities may be required at step 4, and made more explicit in 
step 6, rather than “how has this helped you?” “I’m not sure this should be an extra step or just an 
example but to collect some of the data and evidence, people who are new to evaluating might not 
know that you need ethics approval” (participant 2BSa). This was exemplified in a precursor phase 
of this study where the authors followed three projects through their engagement with project 
evaluation. One of those projects did not obtain ethics clearance and was thus limited in how they 
could disseminate findings (Huber & Harvey 2016b). 

Regarding the dissemination of evaluation findings (Table 3, step 6), two participants (2IS and 2E) 
suggested the inclusion of people who undertake support roles in the institution, such as teaching 
and learning units and the library, both central and Faculty-based, so they can facilitate cross- 
disciplinary learning. “[The central unit] should be interested stakeholders to see what the 
outcomes of my project are for when they give advice to other people requesting grants” 
(participant 2IS). This also aligns with findings from the previous phase of the wider study (Huber 
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& Harvey 2016b). A community of practice was also mentioned not only as a means of 
dissemination but also as a valuable outcome in and of itself. “The outputs.... could also be 
communities of practice or different ways of getting practice sharing and engagement or ongoing 
change” (participant 2E). This reinforces earlier comments on the value of communities of 
practice for building evaluation capacity, another of the findings from phase 1. 

The Theory of Change approach to evaluation is one way of investigating the progress of a 
particular project or intervention. This approach aims to “determine its intended outcomes, the 
activities it expects to implement to achieve those outcomes, and the contextual factors that may 
have an effect on implementation of activities and their potential to bring about desired outcomes” 
(Connell & Kubisch 1998). In this approach, information about the progress of an intervention can 
provide guidance not only that change is required but how and why certain activities can lead to 
improvement. This was raised during the focus group when the participants discussed the benefits 
of formative evaluation: “So you are doing a little bit of evaluation as you are going through 
anyway and you are using the output of that before you complete the project” (participant 2BS). 
Again, this aligns with the concept of evaluation for learning as put forth by Preskill and Torres 
(2000), who suggest that “when individuals participate in an evaluation process that is 
collaborative and guided by dialogue and reflection, learning occurs not only at the individual 
level but also at the team and organization levels” (p. 26). 

Earlier phases of this study found that named evaluation approaches were rarely used (Huber & 
Harvey 2016a). This is not to say evaluation wasn’t happening; rather, it may indicate that many 
practitioners in the area of small-grant projects were unaware of the different evaluation methods 
and approaches. 

The final key learning from this second-cycle focus group was about the language used in the 
framework. One participant said, “I think it would be better to use the language of progress rather 

than achievement. Progress is very valuable” (Participant 2PM). This is a concept endorsed by 

the evaluation scholar Wadsworth (2010) in her work on human inquiry for living systems, which 
suggests that the process of building research and evaluation into everyday actions can generate 
new knowledge in addition to facilitating learning through inquiry. Using such language also has 
the potential to develop a scholarly approach to learning and teaching. 


Reflecting on lessons learned about designing an evaluation¬ 
planning framework 

A number of pertinent recommendations emerged from the findings and discussion sections. When 
designing an evaluation-planning framework that aims to be practical and flexible for a range of 
applications in the higher-education sector, it is recommended to: 

1. Include significant background information about the usefulness of an evaluation¬ 
planning framework to preface the use of the framework and situate it in the context of 
the institution. Exemplars can assist in this (O’Neill 1995). 

2. Limit the number of steps/questions to reduce the perception of time needed to interact 
with the framework (Ryan, Chandler & Samuels 2007). 

3. Include a set of reflective questions to situate each step and further clarify what is 
required (Botcheva, Shih & Huffman 2009; Rog 2012). 
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4. Offer users some pre-set options from which to choose, but also some open-ended options 
for them to add their own contexts. 

5. Allow users some flexibility in which questions they would like to use in the framework, 
such that it is aligned with their context and grant-application or reporting requirements 
(DeLuca et al. 2009; Sheard & Markham 2005). 

6 . Use the language of progress rather than achievement, as smaller grants often use 
evaluation as a monitoring or development checkpoint (Connell & Kubisch 1998; 
Wadsworth 2010). 

These lessons learned have been incorporated into the final SPELT framework (Table 4). The 
author invites colleagues to make use of the SPELT framework to validate and confirm its 
transferability to different contexts. 

Table 4. The SPELT framework including example reflective prompts 


Step 

Action 

Reflective prompts 

1. 

Define the purpose and scope 

• What exactly are you evaluating? 

• Why is the evaluation being done? 

• Are you basing the evaluation on any particular method, 
framework or approach? 

Next consider the results from the evaluation. 

• How will the information be used? This may depend on 
the audience for the evaluation, so you may need to return 
to this question after completing step 2. Considering usage 
may also help narrow down the scope of the evaluation. 

• Who will evaluate this project? Are they suitably skilled? 

• What value will the evaluation process add to the project? 

2. 

Identify the stakeholders 

• Are the stakeholders also part of the study audience? 

• Consider how competing interests should be prioritised. 

3. 

Develop key evaluation questions 

• Are the questions summative or formative? 

• How many questions are reasonable? 

4. 

Collect and analyse data 

• What are the most appropriate methods of data collection? 

• How will any ethical issues be addressed? 

5. 

Define measures of success 

• Consider what it would take for the project to be deemed 
successful. For example, it may be that it produces and 
disseminates information about the intended 

outcomes. 

• What items are considered as failure points? 

6. 

Report or disseminate strategies 
and recommendations 

• What are the functions of reporting? 

• Who will read the reports? 

• When will reporting take place? 

• What kinds of information will be included in evaluation 
reports ? 
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For example, the funding body may need information on 
how resources have been used; formative reporting could 
provide feedback to the project team on processes or design; 
and engaging stakeholders may be required for greater use 
of findings. _ 


Limitations 

Two limitations are noted regarding this study. First, it is situated in the Australian higher- 
education context, and the framework is designed for that context. However, higher-education 
sectors in other countries have similar funding systems supporting the development of learning 
and teaching innovations (Huber & Harvey 2013). Early iterations of the framework and the need 
for such a support mechanism have been trialled with a number of education and evaluation 
practitioners through presentations and focus groups. However, wider dissemination of the SPELT 
framework needs to take place to learn about transferability to the international sector. 

The second limitation of this study is the number of iterations that occurred in the design of the 
SPELT framework. An open-access online version is currently being trialled and invites feedback 
from users. This data will be used to further refine the examples and use-cases for the framework. 


Conclusion and next steps 

The SPELT framework has been developed through three phases of research using primary data 
from learning and teaching practitioners in higher education. Combined with findings from the 
evaluation literature, a six-step approach to evaluation planning has been shown to be an 
appropriate approach in this study for small, internally funded learning and teaching projects. Such 
an approach offers a better fit for purpose than some of the more elaborate approaches and 
frameworks designed for projects with larger budgets. Further research is now needed to test the 
SPELT framework with a wider cohort of users, examine the longer-term gains and ensure that the 
six steps meet the evaluation needs of a diverse range of projects. In addition, the SPELT 
framework could be trialled with larger projects to evaluate its applicable range. 

The research findings also show that if such an evaluation-planning framework is designed to be 
responsive to various contextual requirements, it could act as a potential instigator for reflective 
practice and leading innovative change in learning and teaching. Six recommendations have been 
made that underpin the design and implementation of the SPELT framework; they are 
strengthened by the literature and supporting comments from the participants in this testing phase. 

It must be clarified, however, that a systematisation of evaluation criteria is not being suggested 
here, though this may be an area for future investigation to compare across projects. 

This study has also shown that users want to save time if they have to complete evaluation 
activities. The next step in this study is to trial the use of the SPELT framework in an interactive 
electronic or online format, incorporating the recommendations above. The author has begun this 
process by developing an online version of the framework using form input, datasheets and some 
simple coding to trial its versatility and usefulness with an audience of internally funded learning 
and teaching grant holders in the higher-education sector. 
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Appendix A - An excerpt from a supporting resource for staff 
containing specific, practical and authentic examples to illustrate 
each of the steps in the evaluation planning framework 


Step 

Example (small, internally funded project) 

1. What is the purpose and scope of the 
evaluation? Consider also how the 
information will be used. 

Purpose: 

• Determine if stated objectives are achieved 

• Measure success of project/innovation 

• Document appropriate spending of funds/accountability 

• Measure impact 

Scope: 

• Used to seed a larger grant 

• Support for embedding intervention more widely 

• Shared with colleagues 

2. Who are the stakeholders of the project 
and of the evaluation? Are they also part of 
the study audience? 

Project: 

• Students (current and future) 

• Subject coordinators 

• Tutors in the subject 

• Faculty’s Associate Dean Teaching & Learning 

• Colleagues at other institutions 

Evaluation: 

• Grant committee/funder 

• Colleagues 

3. What are the key evaluation questions? 

Formative: 

• What measures, if any, are being put in place to promote sustainability of the 
project's focus and outcomes? 

• What factors are helping and hindering in the achievement of the outcomes? 
Summative: 

• What lessons have been learned from this project and how might these be of 
assistance to other colleagues/institutions? 

4. What data and evidence will be collected 
and how will it be analysed? 

In a small project (team of one or two), this could consist of: 

• Observation/reflection data 

• Student evaluations of an intervention 

• Grades 

• Feedback from stakeholders and from peers 

Analysis: 

• Qualitative, through thematic coding of responses 

• Quantitative to show improvement 

5. What are the criteria for judgment? 

• Improved student feedback/satisfaction survey results 

• Improved grades 

• Student satisfaction with outcomes 

• Lower attrition rates 

• Access rates 

6 . What dissemination strategies will be 
used and how will this help you? 

• Publications 

• Presentations 

• Running workshops for colleagues 

• Reports to grant body/management 

• Newsletter/blog articles 

• Critical friend 


http://ro.uow.edu.aU/jutlp/voll4/issl/9 


16 





Huber: Learning and Teaching Project Evaluation Planning Framework 


Help: 

• Raise awareness of the intervention 

• Gain feedback for next stage/project 
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